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Let G be a connected Lie group and Aut (G) be the group of all (continuous) 
automorphisms of G. When does the action of Aut (G) on G have a dense orbit? 
It turns out that if this holds then G is a nilpotent Lie group; (see [Q). However 
it does not hold for all nilpotent Lie groups, and a characterization of the class of 
groups for which it holds seems to be a remote possibility. When G is a vector 
space then Aut (G) is its general linear group, and the action has an open dense 
orbit, namely the complement of the zero. On the other hand there exist 3-step 
simply connected nilpotent Lie groups such that every orbit of the action is a 
proper closed subset; this holds for the simply connected Lie group corresponding 
to the Lie algebra described in ||; see also §f| below. Among the simply connected 
groups this brings us to considering the question for 2-step nilpotent Lie groups. 
In this note we construct an example of a 2-step simply connected nilpotent Lie 
group G for which the action of the automorphism group has no dense orbits; see 



Theorem 2.1 



We discuss the question also for Lie groups which are not simply connected, 
and show that while for all connected abelian groups other than the circle the 
automorphism group action has dense orbits, among the quotients of the group G 
as above by discrete central subgroups there are nilpotent Lie groups G' such that 
every orbit of Aut (G') on G' consists of either one or two cosets of the commutator 
subgroup [G',G'], the latter being a closed subgroup; see Corollary |5l2"| . 

The method involved also enables us to give an example of a 3-step simply 
connected Lie group whose automorphism group is nilpotent (its action on the Lie 
algebra is by unipotent transformations); see §|j; the condition implies in particular 
that all orbits of its action on G are closed. An example of a 6-step simply connected 
nilpotent Lie group with this property was given earlier in |J. 

Another motivation for studying the automorphism groups of nilpotent Lie 
groups comes from the question of understanding which compact nilmanifolds sup- 
port Anosov diffeomorphisms and which do not. It is known that if G is a free 
fc-step nilpotent Lie group over a n-dimensional vector space V with k < n, then 
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for any lattice r in G the nilmanifold G/Y admits Anosov automorphisms; see 0, 
pp. 558; see || for another approach to the question. There are also other classes 
of compact nilmanifolds for which this holds; see [|IJ, 0, |7|]. However in general 
compact nilmanifolds may not admit Anosov automorphisms. Clearly, if G is a 2- 
step simply connected nilpotent Lie group such that [G, G] is one-dimensional then 
G/Y can not admit an Anosov automorphism, for any lattice Y . Only a few other 
examples of 2-step simply connected Lie groups are known with this property; see 
0, @, IPH - We shall show that for the 2-step nilpotent Lie group G in our exam- 



ple, if T is a lattice in G then the nilmanifold G/Y has no Anosov automorphism, 
and also no ergodic automorphism; see Corollary |6.1|. 



1 Lie algebras and Lie groups 

We recall that a Lie algebra Q is said to be nilpotent if the central series {C/j} 
defined by Q = Q and Q i+ i = [Q, Q/\ for all % > 0, terminates, namely there exists 
k > 1 such that Qk = 0; if k is the smallest integer for which this holds then Q 
is said to be fc-step nilpotent. We say that a connected Lie group G is a /c-step 
nilpotent Lie group if the Lie algebra of G is a fc-step nilpotent Lie algebra. 

For a Lie group G we shall denote by Aut (G) the group of all continuous 
automorphisms of G, and similarly for a Lie algebra Q we denote by Aut (Q) the 
group of all Lie automorphisms of Q. 

Let G be a connected nilpotent Lie group. Let G (1) = [G, G] and Z{G) denote 
the center of G. Suppose that Z(G) is contained in G^; this condition holds for the 
nilpotent groups discussed in the following sections. Then for any (continuous) ho- 
momorphism ip : G/G*- 1 - 1 — > Z(G) the map r : G — > G defined by r(g) = gip(gG^) 
for all g e G is an automorphism of G; we shall call an automorphism arising in 
this way a shear automorphism. The class of shear automorphisms forms a normal 
subgroup of Aut (G) . Every automorphism r of G factors to an automorphism 
of G/G^; we denote the factor of r on G/G^ by r. Clearly r i— ► r is a homo- 
morphism of Aut (G) into Aut (G/G^). We denote by -4(G) the image of the 
homomorphism, namely, 

A(G) = {Te Aut (G/G (1) ) | r g Aut (G)}. 

We note that for all shear automorphisms as well as all inner automorphisms of G 
the factor on G/G*- 1 - 1 is trivial. 

Suppose now that G is a simply connected nilpotent Lie group and let Q be 
the Lie algebra of G. Then G^ = [G, G], namely [G, G] is closed, and G/[G, G] 
and [G, G] are (topologically isomorphic to) vector spaces; they may be identified 
canonically with Q/[Q,Q] and [G,G] respectively. Also in this case Aut (G) can be 
realised as Aut (Q) identifying each automorphism with its derivative on Q. We 
note also the following: 
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Lemma 1.1. Let G be a simply connected nilpotent Lie group and Q be the Lie 
algebra of G. Let r G Aut(G) and suppose that T is unipotent (as an element of 
GL(Q/[Q,Q])). Then r is unipotent (as an element of GL(Q)). 

Proof. It can be seen that the largest r-invariant subspace, say Go, of Q on which 
the restriction of r is unipotent (namely the generalized eigenspace for 1 as the 
eigenvalue) is a Lie subalgebra of Q. Since f is unipotent Go + [G, G] = G- Substi- 
tuting from the equation for G on the left hand side successively and using that G 
is nilpotent we deduce that Go = G- Therefore r is unipotent. 

Now let V be a (finite dimensional) vector space and let W be a subspace of 
A 2 V, the second exterior power of V. We can associate to these canonically a 
2-step nilpotent Lie group as follows. Let V = (A 2 V)/W and Q = V © V. We set 
[vi, v 2] = V\ A v 2 mod W for all Vi,V2 G V, and [x, y] — for all x G G and y G V . 
These relations extend uniquely to a Lie bracket operation on G- Furthermore 
G is a 2-step nilpotent Lie algebra, with G/[G,G] = V. Conversely every 2-step 
nilpotent Lie algebra G can be realised as a Lie algebra associated to a datum as 
above, with V = G/[G, G], and W a suitable subspace of A 2 V. 

Let G be the simply connected Lie group corresponding to the Lie algebra G 
associated to a pair V, W as above. Then G/[G, G] may be realised canonically as 
V; this identifies the subgroup A{G) with a subgroup of GL(V). It is easy to see 
that x G GL(V) is of the form r for some r G Aut (G) if and only if the subspace W 
of A 2 V is invariant under the action induced by x on A 2 V. It follows in particular 
that A(G) is an algebraic subgroup of GL(V). 

We note also the following: 

Proposition 1.2. Let G be a simply connected nilpotent Lie group. Then every 
orbit of Aut (G) on G is open in its closure. In particular every dense orbit is open. 
Consequently, if the A(G) -action on V has no open orbit then the Aut (G) -action 
on G has no dense orbit. 

Proof. Let G be the Lie algebra of G. Then Aut (G) is an algebraic subgroup of 
GL(G) and this implies that the orbits of its action on G are open in their closures 
(see 0). Since G is simply connected and nilpotent the exponential map is a 
diffeomorphism of G to G and it is equivariant under the actions of the respective 
automorphism groups. Therefore for the action of Aut (G) on G also every orbit 
is open in its closure. The second assertion is immediate from the first. The 
last assertion follows from the fact that the ^4(G)-action on V is a factor of the 
Aut (Cr)-action on G. 
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2 A representation 

In this section we describe a construction of a representation and prove some prop- 
erties. The results will be used in the later sections to give examples of nilpotent 
Lie algebras. 

Let S be the vector space of 3 x 3 symmetric matrices with real entries. For 
each k, I = 1,2,3 we denote by E k i the 3x3 matrix in which the (k, /)-entry (in 
the kth row and Zth column) is 1 and all other entries are 0. Let V be the subspace 
of S defined by 

V = {Ea k iE k i G S | o- 22 = 0"13 + 0"31 = 2o"i 3 }. 

Let <7i, . . . , 05 be the elements defined as follows, forming a basis of V: 

o\ = 2£ii, 02 = E12 + E21, 03 = -^13 + 2-^22 + ^31? °4 = £'23 + -E'32, and 05 = 2E'33. 

Let W be the subspace of A 2 V spanned by the three elements 01 A 04 — 02 A 03, 
0i A 05 — 2 A 04 and 2 A 05 — 03 A 04. 

Let 5, u + and v~ be the matrices defined by 

(2 \ /0 1 0\ /0 0\ 

5= , i/+ = 1 , and v~ = 1 01. 
\0 -2/ \0 0/ \0 1 0/ 

The space spanned by the three matrices is a Lie subalgebra of the Lie algebra of 
3x3 matrices of trace 0, namely the Lie algebra of SL(3, M). We denote the Lie 
subalgebra by Ti. We note that it is isomorphic to the Lie algebra of SL(2,M), 
with 5, u + and v~ corresponding to the standard basis of the latter; in particular 
the subspace spanned by 5 is a Cartan subalgebra. Let H be the connected Lie 
subgroup of SL(3, M) corresponding to Ti. Consider the action of H on IB? given 
by restriction of the natural action of SL(3,M) on M 3 . We see that there is no 
proper nonzero subspace on M 3 invariant under both exp z/ + and exp v~ . It follows 
therefore that the action of H on M 3 is irreducible. Since H contains an element, 
viz exp 5, with distinct eigenvalues this implies that every element of the center of 
H acts by scalar multiplication by real numbers, and considering the determinant 
we see that it must be trivial. Thus H has trivial center. This shows that H is Lie 
isomorphic to PSL{2, M), the adjoint group of SL(2,M). 

Consider the action of SL(3, M) on S given by (x, 0) 1— > xcrx 1 for all x G 
SL(3,1R) and e S, where x l denotes the transpose of x. It is straightforward 
to verify that the subspace V is invariant under the action of H (obtained by 
restriction); it suffices to verify that V is invariant under the corresponding action 
of the Lie algebra, given by (£,0) 1— > £0 + 0^, for all £ G 7i and G S, and 
furthermore it is enough to consider £ = 5, u + and v~ '. The elements 01, . . . , 05 are 
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weight vectors with respect to 5 with weights 4, 2, 0, —2 and —4 respectively, and 
the latter being distinct shows that the representation of H over V is irreducible. 

We shall realize if as a subgroup of GL(V) by identifying each h in H with 
its action on V (this is indeed an injective correspondence). We denote by D the 
one-parameter subgroup of GL(V) corresponding to the one-parameter subgroup 
{exptS | t e M}. We note that for any nontrivial element of D, a±, ...,05 are 
eigenvectors with distinct eigenvalues, and hence the centralise of D in GL(V) 
consists of diagonal matrices with respect to the basis 01, . . . , 05. 

Let A be the subgroup of consisting of all x in GL(V) such that W is invariant 
under the action induced by x on A 2 V. A straightforward computation shows that 
H (viewed as a subgroup of GL(V)) is contained in A. We note that A is an 
algebraic subgroup of GL(V). Let A u be the unipotent radical of A, namely the 
largest normal subgroup consisting of unipotent elements. Then the set of common 
fixed points of the action of A u on V is a nonzero subspace invariant under H, and 
since the if-action on V is irreducible it follows that A u fixes all points of V, which 
means that A u is trivial. Therefore A is reductive. 

Theorem 2.1. A = ZH, where Z is the subgroup of GL(V) consisting of scalar 
multiplications by real numbers. The A-action on V has no open orbit. 

Proof Let A be the connected component of the identity in A and let S = [A , A }, 
the commutator subgroup. As A is reductive it follows that S is a semisimple 
subgroup of GL(V); (see flQfl , Ullf ). Since if is a simple Lie subgroup of A it 



follows that it is contained in S. We shall show that S = H. Firstly let C be the 
maximal compact connected normal subgroup of S. Since S is semisimple, there 
exists a closed connected normal subgroup S' such that S = S'C and 5" PI C is 
a finite subgroup contained in the center of S; both 5" and C being connected 
subgroups this implies in particular that every element of S' commutes with every 
element of C. Since if is a noncompact connected simple Lie group it has no 
nontrivial homomorphism into a compact Lie group. Applying this to the quotient 
homomorphism of S onto S/S' = C/(C fl S') we conclude that H is contained 
in S'. Therefore C is contained in the centralizer of D. Since the centralizer is 
diagonalizable and C is a compact connected subgroup it follows that C is trivial. 
Thus S has no compact normal subgroups of positive dimension. 

Now recall that the subspace W of A 2 V is invariant under the action of A and 
in particular that of S, and let (p : S — * GL(W) be the representation of S induced 
by the action. Let K be the connected component of the identity in the kernel of 
ip. Then K is a connected normal subgroup of S. Let A be a maximal subgroup 
of S such that D is contained in it and its adjoint action on the Lie algebra of 
S is diagonalisable over M. Then A intersects (by Aswan decomposition) any 
noncompact connected normal subgroup nontrivially, and since S has no compact 
normal subgroups it follows that A fl K is nontrivial, unless K is trivial. Let 
a G A H K. Then a commutes with all elements of D, and hence 01, . . . , o~ 5 are 
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eigenvectors of a; let Ai, . . . , A5 be the corresponding eigenvalues. Since a G K the 
action induced by a on W is trivial. This means that the vectors o\ A a 4 — 02 A 03, 
G\ A er 5 — cr 2 A <t 4 and <j 2 A cr 5 — cr 3 A 04 are all fixed under the action of a. Each 
crj A (jj is an eigenvector of the action of a on A 2 V and hence the three vectors as 
above being fixed implies that the vectors U\ A 04, 01 A 05, er 2 A cr 3 , 2 A 04, cr 2 A 05 
and 03 A 04 are fixed individually. Therefore for the eigenvalues we get the relations 
A1A4 = A1A5 = A2A3 = A2A4 = A2A5 = A3A4 = 1, which is possible only if Aj = 1 for 
alH = 1, . . . , 5. Since a is diagonalizable this shows that a is the identity. Thus 
A H K is trivial and hence, as noted above, K is trivial. Therefore the kernel of tp 
is discrete. 

Clearly <p(S) is a connected semisimple Lie subgroup of GL(W) containing 
(f(H). Since W is 3-dimensional this implies that (p(S) is either <p(H) or SL(W); 
(inspection of the weights for the action of (p(D) on the Lie algebra of SL(W), 
via the adjoint representation of SL(W), shows that the y?(ii)-action has only 
two irreducible components, one of them being the Lie subalgebra of (p(H) itself; 
this means that there is not even a subspace invariant under the y>(i?)-action lying 
strictly between the Lie algebras of <p(H) and SL(W), which in particular implies 
the assertion here; the assertion can also be proved in other ways). Since ip has 
discrete kernel this shows that the Lie algebra of S is isomorphic to that of either 
H or SL(3,M). We note that the Lie algebra of SL(3,M) has no irreducible 5- 
dimensional representation; this can be deduced from the corresponding statement 
over the field of complex numbers, the latter being easy to see from the classification 
theory of representations of semisimple Lie algebras; see |[13|| . Recall that the in- 
action on V is irreducible. Since S contains H, its action on V is irreducible, and 
since V is 5-dimensional the preceding observation implies that the Lie algebra of S 
can not be isomorphic to that of SL(3, M). Therefore from the alternatives above 
we now get that S is locally isomorphic to H. Since S is connected and if is a 
subgroup of S this implies that S = H. 

Since H = S = [A , A }, in particular H is normal in A. Now let a G A be 
arbitrary and consider the automorphism of H induced by the conjugation action 
of a. Since H is Lie isomorphic to PSL(2, M) every automorphism of H is inner. 
Hence there exists ho G H such that aha' 1 = h^hh^ 1 for all h G H. Then hg 1 a 
commutes with every element of H and particular with all elements of D. Therefore 
h,Q l a is diagonalisable and in particular all its eigenvalues are real. Since the action 
of H on V is irreducible it now follows that h$ 1 a acts by scalar multiplication by a 
(nonzero) real number. Thus h$ 1 a G Z in the notation as in the hypothesis. Then 
a G HZ = ZH, and since a E A was arbitrary we get that A is contained in ZH. 
On the other hand H and Z are contained in A and therefore we have A = ZH. 
This proves the first assertion in the theorem. The second assertion follows from 
the fact that ZH is 4-dimensional while V is 5-dimensional. 

We note also the following simple fact about representations of PSL(2, M): 
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Lemma 2.2. Let p : PSL(2,M) — > GL(Q) be a representation of PSL(2, M) over 
a M-vector space Q of dimension n > 5, such that no nonzero point of Q is fixed by 
the image of p. Let E be the set of points p inQ such thatp is an eigenvector of p(g) 
for some nontrivial g G PSL(2, M). Then E is a union of countably many smooth 
submanifolds of dimension at most (n + 4)/2. In particular if L is a subspace of Q 
of dimension m > (n + 4)/2 and A is the Lebesgue measure on L then X(EnL) = 0. 

Proof. We realize PSL(2, M) as the group H as above (this is just for notational 
convenience). For each t G 1R let d t — expt5 and u t = exptz/ + , where 5 and v + are 
the matrices as defined earlier. Let {kt} be a periodic one-parameter subgroup of 
H, say with period 2ir. Then every element of H is conjugate to one of d t , u t or k t 
for some t G M. Let R be the countable subset of H consisting of d\,ui, k\ and k r 
for all r of the form 2-njm with m a positive integer. 

For any h G H let E(h) denote the set of points in L which are eigenvectors of 
p{h). Then E(h) is a union of finitely many vector subspaces of L; we shall denote 
by d(h) the maximum of the dimension of the subspaces contained in E(h). Now 
consider any nontrivial element h in H . We can find a conjugate h! of h such that 
the subgroup generated by h and h! is dense in H = PSL(2, 1R). Then every point 
of E(h) fl E(h') is an eigenvector of all elements of p(H), and since the latter is a 
simple Lie group the points must be fixed by p(H). The condition in the hypothesis 
therefore implies that E(h)C]E(h') = 0. Since h and h! are conjugates d(h) = d(h') 
and hence the preceding conclusion implies that d(h) < n/2. 

Now let p G E and h G H be such that p is an eigenvector of p(h). Suppose 
first that h is of infinite order. There exists t G M and g G H such that ghg" 1 
is one of d t , u t or k t . Then p{g){p) is an eigenvector of d t , u t or k t . This implies 
that it is an eigenvector of one of the elements d±, U\ or k\ respectively; in respect 
of the last alternative note that since k t is of infinite order the closure of the 
cyclic subgroup generated by it contains k±. Thus there exists x G R such that 
p(g)(p) G E(x), and so p G p{H)E(x). Now suppose that h as above is of finite 
order. Then h is conjugate to k r for some r of the form 2-njm for a positive integer 
m. Therefore in this case also we get that p(g)(p) G E(x) for some x G R. Thus 
E = U xe R p(H)(E(x)). Each E(x) is a finite union of vector subspaces which are 
invariant under a one-parameter subgroup of H and have dimension at most n/2. 
It follows that each p(H)(E(x)) is a finite union smooth submanifolds of dimension 
at most (n/2) + 2 = (n + 4)/2. This proves the Lemma. 

In respect of the conclusion as in the lemma the situation in low dimensions 
is as follows. For dimensions 2 and 4 there are no representations of PSL(2, M) 
with no nonzero common fixed points. In dimension 3 the realization of H (as 
above) as a subgroup of SL(3, M) is (up to equivalence) the only representation 
with no common fixed points. For this representation the set of points which are 
eigenvectors of nontrivial elements form a cone in M 3 with nonempty interior. 
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3 A 2-step nilpotent Lie group 



We now use Theorem ^J] to give an example of a simply connected 2-step nilpotent 
Lie group G such that the action of Aut (G) on G has no dense orbit. 

Let V and W C A 2 V be as in the preceding section and let Q be the 2- 
step nilpotent Lie algebra associated with the pair V, W as in §|]; namely Q = 
V © V, where V = (A 2 V)/W and the Lie bracket operation is determined by the 
conditions [t>i,t>2] = (^i A f 2 ) (mod W) for all Vi,v 2 G V, and [x,y] = for x G Q 
and y G V. Let G be the simply connected nilpotent Lie group corresponding to 
Q. Then we have 

Corollary 3.1. The action of Aut (G) on G has no dense orbit. 

Proof. For the Lie algebra Q as above, corresponding to G, the subgroup A{Q) 
as defined in §0 is the same as the subgroup A as in Theorem |2J]. Since by 
Theorem [O] the action of A on V has no open orbit, by Proposition [TJ the 
Aut (G)-action on G has no dense orbit. 



Using Theorem |2.1| one can also describe Aut (G) as follows. To each h G H 
corresponds a Lie automorphism of Q such that the restrictions to V and V = 
(A 2 V)/W are respectively given by the /i-action on V and the factor of the h- 
action on A 2 V. We denote by H the group of automorphisms of G for which 
the associated Lie automorphism corresponds to some h G H as above. Then 
if is a Lie subgroup of Aut(G), canonically Lie isomorphic H. Similarly each 
z G Z corresponds canonically to an automorphism of G, and we shall denote by 
Z the subgroup of Aut (G) corresponding to Z. Also let ^ be the group of shear 
automorphisms of G (see §[!]). 

Corollary 3.2. Aut(G) = (ZH)ty, semidirect product (with ^ as the normal sub- 
group). 

Proof. We note that (ZH)ty is a subgroup of Aut (G). Now consider the homo- 
morphism r i— ► r of Aut (G) into GL(V). It is easy to see that its kernel is and 
its image is A which by Theorem [2.1| is ZH. As the subgroup (ZH)ty of Aut (G) 
contains and its image under the homomorphism as above is ZH, this shows 
that it must be the whole. Since the map r i— > r is injective on ZH it follows that 
{ZH ) n ^ is trivial. This proves the corollary. 

Remark 3.3. The action of H on A 2 V (notation as before) decomposes into two 
irreducible components, one being on the subspace W as above and another on 
the subspace, say W, spanned by the if -orbit of o\ A <r 2 ; this can be deduced by 
inspection of the set of weights. Let Q' be the 2-step nilpotent Lie algebra associated 
to the pair V and W, as described in §|I[ Then with some modifications in the 
above argument it can be shown that Aut {G') has no open orbit on G'\ in this 
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respect we note mainly that W is also invariant, along with W, under the action of 
A(G') on A 2 V. This yields an 8- dimensional example (as against 12 of Q) for which 



the automorphism group action has no dense orbit. The example as in Theorem pTT 
on the other hand admits a simpler presentation. 



4 Lie groups with nilpotent automorphism groups 

An example of a 6-step simply connected nilpotent Lie group such that Aut (G) 
is nilpotent was given in ||. Earlier in || an example of a 3-step nilpotent Lie 
algebra Q was constructed for which all derivations are nilpotent; the condition 
implies that Aut (G)°, the connected component of the identity in Aut (Q), consists 
of unipotent elements (viewed as elements in GL(Q)), and hence is a nilpotent 
Lie group; Aut (Q) itself is however not nilpotent for that example, as has been 
remarked in H . In this section we shall describe a class of examples of simply con- 
nected 3-step nilpotent Lie groups, related to the results in the preceding sections, 
for which the all automorphisms are unipotent, and the automorphism groups are 
nilpotent. The following implication of this to orbits of the automorphism group 
action may be borne in mind. 

Remark 4.1. Let Q be a nilpotent Lie algebra such that Aut (Q)° consists of 
unipotent elements. Then Aut (Q)° is a unipotent algebraic subgroup of GL(Q) and 
hence all its orbits on Q are closed (see [jl(J). Since Aut (£?) is an algebraic group it 
has only finitely many connected components and hence we get furthermore that 
all orbits of Aut (Q) on Q are closed. As the Aut (G)-action on G and the Aut (en- 
action on Q are topologically equivalent (via the exponential map) it follows that 
all orbits of Aut (G) on G are closed. Thus in this case we have a situation stronger 
than the orbits not being dense, that was obtained in §[| 

We note also the following, which shows that the 3-step condition in the above 
assertions is optimal. 

Remark 4.2. Let G be a 2-step simply connected nilpotent Lie group. Then for 
the action of Aut (G) on G the closure of every orbit contains the identity element; 
in particular orbits other than that of the identity element are not closed; (in this 
case the automorphism group, or even its connected component of the identity, 
does not consist entirely of unipotent elements (viewed as automorphisms of the Lie 
algebra). The assertion is immediate from the fact that all scalar transformations 
of V = G/[G,G] belong to A(G); this shows that the closure of every orbit of 
Aut (Q) on Q, where Q is the Lie algebra of G, contains the zero element of Q\ this 
is equivalent to the statement as above. 

We now proceed to describe the 3-step nilpotent Lie algebras. We shall follow 
the notation as in §0. For 1 < i < j < 5 let Pij — &i A Gj mod (W). We note that 
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Pu = P23, Pis = P24, andp 2 5 = P3A, and that {p 12 , p 13 , Pu, Pis, P25, P35, P45} is a basis 
of V. Let L be the subspace of V spanned by {^13,^14,^15,^25,^35,^45}- Then by 
Lemma [2.2| there exists (a nonzero element) p G L which is not an eigenvector of 



the action of any nontrivial element of H (under the if-action on V) . Now we set 
Af = V © V and we define a Lie bracket operation by the relations [<jj, <x,] = a, t A <jj 
(mod W) for all i,j = 1,...,5, [x,x'] = for all x,x' G V, [ciyPn] = P, and 
[&k,Pij] — if either k 7^ 1 or 7^ (1,2); the product defined on the basis 

elements extends uniquely to a Lie bracket operation on A" as above; we note in 
particular that by definition [<jj, [tfj, o^]] = if i,j and k are distinct indices, from 
which one can see that the Jacobi identity holds for the product. Furthermore, 
since p is contained in the subspace L as above it follows that p is contained in the 
center of Af. Since p spans [Af, [Af, Af]] this shows that Af is a 3-step nilpotent Lie 
algebra. Let N be the simply connected (nilpotent) Lie group corresponding to M . 
We prove the following: 

Corollary 4.3. The subgroup *4.(iV) consists only of the identity element. In par- 
ticular Aut(N) consists of unipotent elements, when realised (canonically) as a 
subgroup of GL(J\f) ■ 

Proof. Let r G Aut (A/"). Then r factors to a linear transformation x = f G GL(V). 
Since [a i: <jj] = Oi A Oj (mod W) for all i,j = 1, ... ,5, it follows that the action 
of x on A 2 V leaves invariant the subspace W. Hence by Theorem |2.1| x G ZH . 
Let z G Z and h G H be such that x = zh. Since the subspace spanned by p 
equals [Af, [Af, Af]} it is invariant under all automorphisms of Af. Thus p is an 
eigenvector of x (for the action on V). Since the elements of Z act as scalars on 
V , it follows that p is an eigenvector of h. Since by choice p is not an eigenvector 
of any nontrivial element of H, we get that h is trivial. Thus r = x = z, a scalar 
transformation, say multiplication by A. Then the action of r on V is given by 
multiplication by A 2 . Since [o"i,pi 2 ] = p, we have X 2 p = r{p) = r([cr 1 ,p 12 ]) = 
[t(cti), r(pi 2 )] = A 3 [cri,pi 2 ] = A 3 p. Since p is a nonzero element this implies that 
A = 1. This means that z is trivial, and therefore r is trivial. Thus r is trivial 
for all r G Aut (AT). This shows that *4(AQ is trivial. The second assertion follows 
from this, together with Lemma Thus proves the corollary. 



We deduce also the following, showing that under a slight further condition the 
automorphism group is 'minimum possible'. 

Corollary 4.4. Let N and Af be as above. Suppose the point p in the definition 
of the Lie algebra structure on Af is not contained in the subspace spanned by 
{Pu,Pi5,P25,P35,P45} ■ Let N* denote subgroup of Aut (N) consisting of all inner 
automorphisms and ^ be the subgroup consisting of all shear automorphisms of N. 
Then Aut(N) = N*^ . 

Proof. We shall identify Aut (N) with Aut (Af) as before. As an algebraic subgroup 
consisting of unipotent elements Aut (Af) is a connected Lie group. The Lie algebra 
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T> consisting of all derivations of M is the Lie algebra of Aut (A/"). The subgroups 
iV* and \1/ are normal Lie subgroups of Aut (A/") . Therefore to prove the corollary 
it suffices to show that the Lie subalgebras of T> corresponding to N* and \1/ span 
T>. The Lie subalgebra corresponding to \1/ consists of all derivations 5 such that 
<5 (AT) is contained in L. We shall show that this Lie subalgebra and adoi and a,da 2 
together span Dasa vector space. This would complete the proof. 

Let 6 be any derivation of N '. In view of Corollary [O] its factor on A/"/ [A/, A/] is 
trivial, and hence S(J\f) is contained in V . Hence there exist a\, a 2 G M such that 
5(ai) G ciiPi2 + L, i = 1, 2. Then 5' = 5 — a 2 (ad<7i) + a 1 (ada 2 ) is a derivation such 
that 6' (ai) , 5' fa) G L. It suffices to show that 5' belongs to the Lie subalgebra of 

namely that 5'(J\f) is contained in L. Changing notation we shall assume that 
6(<Ti), 5(02) G L and deduce that S(J\f) C L. 

As 5 (AT) C V, for i,j>2 we have = 5([<Xi, Oj]) = [dai, <jj\ + [a*, <5cr,] = 0. 

Since p i4 = p 23 and pi 5 = p 24 , it follows that S(pu) = = 0. We note 

that <J(pi 2 ) = S([a 1 ,a 2 }) = [Scr 1 ,a 2 ] + [cri,5cr 2 ] G [A", [A", A/]] C L. Therefore 
S(p) = P12]) = [5o"i,Pi2] + [cij^Pi2] = 0. Let L' be the subspace spanned 

by {pu, P15, P25, P35, P45} ■ Then from what we have seen 5(x) =0 for all a; G V . 
By the assumption on p there exists A ^ such that p = Xpi^ + x for some 
x G L'. As 5(p) = and <5(x) = 0, this implies that ^(^13) = 0. Thus we have 
[01, S(aj)] = 8([ai,(jj]) = 5(pij) = for j = 3,4,5. This shows that 5(<Tj) G L for 
j = 3,4,5. Since by assumption 5(ai), 5 {a 2) G L, we now have Sfa) G L for all 
i — 1, ... ,5. Therefore 5(Af) is contained in L. This completes the proof. 



5 Non-simply connected Lie groups 

In this section we shall discuss the analogous questions for Lie groups which are 
not necessarily simply connected. We begin with the following observation. 

Proposition 5.1. Let G be a connected abelian Lie group. Then the action of 
Aut(G) on G has a dense orbit if and only if G is not (topologically isomorphic to) 
the circle group. 

Proof. It is well known that for m > 2 the m-dimensional torus admits automor- 



phisms with dense orbits (see [15] for instance). Now let G — V x C, where V is 
a vector space of dimension n > 1, C is the torus of dimension m > 0. Let TC be 
the set of all continuous homomorphisms of V into C. For each r G GL(V) and 
tp G 7i we get a continuous automorphism of G defined by (v , c) 1— > (r(v),c0(v)) 
for all t> G V and c G C. The automorphisms arising in this way form a subgroup 
of Aut (G) (it is in fact the identity component of the latter). We recall that there 
exist homomorphisms ip : V — > C such that if>(V) is dense in C. Using this it is 
straightforward to verify that under the action of the group of automorphisms as 
above the orbit of any element of the form (v, c) with v 7^ is dense in G. In 
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particular the Aut (G)-action on G has dense orbits. Finally, the circle group has 
only two automorphisms and hence in this case the Aut (G)-action has no dense 
orbit. This proves the proposition. 

For Lie groups G of the form M n x TT m with m/1 there exist abelian subgroups 
of Aut (G), with finite rank, whose action on G has dense orbits; see f| for more 
precise results in this respect. 



In the context of the results in the last section and Proposition |5.1| one may ask 
whether there exist connected 2-step nilpotent Lie groups G such that all orbits of 
the Aut (G)-action on G are closed. By Remark such a group is necessarily non- 



simply connected. The following corollary shows that there are groups with this 
property among quotients of the group G as in §|3|, by discrete central subgroups. 

Corollary 5.2. Let G be the simply connected 2-step nilpotent Lie group as in 
Corollary \3.\ . Then there exists a subset of E of Lebesgue measure in [G, G] 
such that if 9 G [G,G], 9 ^ E and is the cyclic subgroup of G generated by 9, 
then for the Lie group G' = G/0 the following holds: for any automorphism r 
of G' the factor of r on G'/[G',G'] is ±7 , where I is the identity transformation; 
furthermore, orbits of the action of Aut{G') on G' consist of either one or two 
cosets of[G',G'] in G' . 

Proof. Let H be the group of automorphisms as in § |3] and consider the if-action 
on [G, G] . Recall that the latter may be viewed as a vector space and that the 
action is irreducible; in particular there is no nonzero point fixed by the whole 
of H. Since H is Lie isomorphic to PSL(2, M) and [G, G] is 7-dimensional, by 



Lemma |2.2| there exists a subset E of Lebesgue measure such that points outside 
E are not eigenvectors of any nontrivial element of H. Now let 9 G [G, G] , 9 ^ E 
and be the cyclic subgroup generated by 9; since [G, G] is central in G, is 
a normal subgroup. Let G' = G/Q. Since 9 is discrete, the automorphisms of 
G' are precisely factors on G' of automorphisms r of G such that r(O) = 0. We 
note that the shear automorphisms, namely those from ^ in the notation as in §|| 
satisfy the condition. Therefore by Corollary |3.2| Aut (£?') is the semidirect product 
of ^ with the subgroup {r G ZH | r(0) = 0}. Now let r = zh G ZH , where 
z G Z and h G H , be such that t(0) = 0. Since z acts by scalar multiplication 
on [G, G], this implies that 9 is an eigenvector of h. Since 9 ^ E it follows that 
h is the identity element. Hence r = z G Z, and since r(0) = it follows that 
the corresponding scalar transformation of V = G/[G,G] is ±7, where I denotes 
the identity transformation. This implies the first assertion in the Corollary. The 
second one follows from this together with the fact that automorphisms from \l/ 
factor to G'. 
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6 Anosov automorphisms 



As mentioned in the introduction study of the automorphism group can be applied 
also to the general question of understanding the class of compact nilmanifolds 
supporting Anosov automorphisms; see 0, @, @- In this respect we note the 
following consequence of Theorem |2.1|. 



Corollary 6.1. Let G be the 2-step simply connected nilpotent Lie group as in 
Theorem and V = G/[G,G] (as before, realised as a vector space). If t G 
Aut (G) and the factor f on V has determinant 1 then f has a nonzero fixed point 
on V. Consequently, if Y is a lattice (a discrete cocompact subgroup) in G and 
t G Aut (G) is such that t(Y) = Y then the factor automorphism 7t(t) : G/Y — ► G/Y 
is not an Anosov automorphism, and furthermore it is not ergodic. 



Proof. We follow the notation as before. By Theorem [2.1| r G ZH , and furthermore 
if its determinant is 1 then r G H . The first assertion therefore follows from the 
fact that for the if-action on V every element of h has a nonzero fixed point in 
V; for hyperbolic elements this follows from consideration of weights, for parabolic 
elements by unipotence, and for elliptic elements by odd-dimensionality of V (anal- 
ogous assertion holds for any irreducible representation of H on an odd-dimensional 
vector space). 

If T is a lattice in G then [G,G]r is closed and [G,G]Y/[G,G] is a lattice in 
V = G/[G, G]] see [O. Therefore for any r G Aut (G) such that t(Y) — Y, r has 



determinant ±1. Hence by the first part r 2 has nonzero fixed points in V. This 
implies that the factor of r on G/Y is not an Anosov automorphism, and also that 
it is not ergodic. This proves the Corollary. 

Compact nilmanifolds covered by the 3-step simply connected nilpotent Lie 
group G as in §^ also do not support Anosov automorphisms; this is immediate 
from the fact that there are no hyperbolic automorphisms, as [G, [G, G}} is one- 
dimensional. 

Acknowledgement: The author would like to thank Karel Dekimpe for useful com- 
ments on an earlier version of this manuscript. 
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